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measures of performance; (2) balanced attention to implementation and impact; 
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the evaluation; and (5) ways to help a district enhance its IT capacity. Many 
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Motivated by national and state requirements and other incentives, most school districts have prepared an 
information technology (IT) plan and begun its implementation. Because these plans age as quickly as 
the technologies they address, many districts update them yearly. Most do so, however, with little or no 
data on the effectiveness of their efforts. Indeed, despite the detail and dynamic complexity (Senge, 

1990) of bringing IT into the schools, few districts even systematically document their implementation 
efforts. As Education Week(Zehv, 1998) and others have noted, evaluation data on the impact of 
technology on student learning remains quite ambiguous, providing meager justification for the millions 
schools are spending on technology (Postman, 1995; Stoll, 1995; Oppenheimer, 1997; Bryson, 1998; 
Bloom, 1999). 

This situation is not likely to change unless districts incorporate sound evaluation designs into their IT 
plans and successfully execute them. The real benefit of planning, after all, is in the learning that it 
catalyzes. Districts that evaluate their technology plans and initiatives can learn more and better about 
how to refine and redesign them. To capture that learning, districts need to develop and implement 
processes for 1) ongoing collection of data on key variables and indicators, 2) timely communication of 
findings to key decision-makers, and 3) effective use of new knowledge to guide their decisions regarding 
improvement and redesign. The challenge is to help districts craft and use rigorous evaluation systems. 

This paper describes learning from many years of helping districts design IT evaluation systems. This 
work, almost always done as part of developing the strategic technology plan itself, has been conducted in 
dozens of districts of all sizes, including most recently New York City (Center for Educational Leadership 
and Technology, 1997), Philadelphia (Center for Educational Leadership and Technology, 1998), and 
New London, Connecticut (Technology Applications Associates, 1999). Our particular interest is in how 
IT evaluation is viewed, designed, and conducted in districts, what impediments and facilitators exist, and 
how district practitioners might address them. Since much of this work is still under way, our 
observations and conclusions are preliminary. Nevertheless, these findings help to shape a line of inquiry 
and identify variables that merit a more focused examination. This report, therefore, marks a transition 
from a broad exploration of key factors in a complex system to a more analytical approach addressed to 
specific variables. 

I begin with a review of selected contextual variables that impinge on IT evaluation in districts and 
schools and then provide a brief description of selected features, components, and processes of the IT 
evaluation system. The principal focus of this report, however, is on what we are learning about how 
districts evaluate what they are doing with technology. This learning has become increasingly more 
focused during the last few years and has provided us with an opportunity to learn up close about how 
districts can successfully address IT evaluation. The final section presents some implications for practice, 
policy, and research. 



The District Context 

Attempting to understand how districts address the evaluation of their IT initiatives requires an 
appreciation of the broader context in which these actions take place. We have identified five contextual 
variables as important. 
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1. District evaluation capacity. Except in large districts, there is seldom a substantial organizational 
capacity to design and conduct program evaluation. When evaluation is required, it is often 
procured from external organizations or individuals. If internal, it is typically applied to collecting 
and analyzing student test data and not to designing and undertaking comprehensive program 
evaluations, particularly over a multi-year period. Few districts allocate substantial resources to 
internal or external program evaluation. Most do not address standards for evaluation (Sanders, 
1994) or even for systematic learning (Senge, 1990; Garvin, 1993; Schein, 1993). Thus, it is not 
uncommon to observe exemplary practices in one school that are undocumented and sometimes 
unknown in other schools in the same district. Despite the recent interest in school-based action 
research (Glantz, 1998), most districts do not have the resources to conduct formal research. Few 
districts have developed a comprehensive action research function and the organizational capacity 
to implement it. 

2. District use of research and evaluation. Even when research results and evaluation information 
are available, decision-makers at every level — teachers, administrators, and school board 
members — do not regularly use such information to guide their work. There is neither a well- 
established culture for data-based decision support in school districts and schools nor are most 
administrators skilled in transforming performance data into useful reports for teachers and 
stakeholders (Berliner, 1998; Streifer, 1999). Although recently promulgated standards for school 
leaders address these important competencies and dispositions (CCSSO, 1996), most principal 
preparation programs have yet to augment their curriculums in this area. The absence of detailed 
documentation and evidence of what works handicaps decisions about replication and scaling. 
Moreover, the historical boundaries between researchers and practitioners impede productive and 
seamless linkages between knowledge development in higher education and research 
organizations and schools (Hargreaves, 1996). It does not help that few schools of education and 
school leadership programs have thus far served as models of technology use. 

3. Evaluation provisions in IT plans. Although most planning models and guidelines include 
attention to evaluation (e.g., Lumley & Bailey, 1993; Massachusetts Software Council, 1994; 
NSSE, 1996), most district IT plans give little or no attention to documentation and evaluation. 
Objectives are usually not stated as outcomes and in measurable terms. Indicators and 
performance measures are seldom stated clearly enough to guide data collection and analysis. 
Districts expend most of their IT resources on designing and building the IT infrastructure and 
training staff; there are few resources left for monitoring and evaluation. 

4. Critical mass. As Alan Kay and others remind us, the technology revolution has hardly begun 
(Kay, 1998; Negroponte, 1995). The critical mass of telecommunications infrastructure, 
hardware and software, trained teachers, and technology-rich student learning opportunities 
needed to produce and sustain an impact is not yet available in most districts (NCES, 1998). 
Although there are varying opinions about what constitutes critical mass (Moursund, 1999), it is 
clear that the scope of the IT intervention thus far is of insufficient mass and so primitive to 
render premature evaluation judgments about impact on student learning (Glennan & Melmed, 

1 996). As Green states, “It is still premature to talk about a technology-driven transformation of 
educational institutions because virtually all schools and colleges are still in the early stages of 
adopting and incorporating various kinds of IT resources into their instructional functions. And it 
is hyperbole to discuss a technological revolution in education, which implies a sudden and 
dramatic departure from past practices — practices that reflect, in part, academic traditions that are 
centuries old (1999, p. 12).” 

5. Multiple decision-makers. School leaders must inform multiple audiences about student, 
program, and organizational performance. Such is the case with technology applications. Each 
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audience defines success with different indicators. For example, many stakeholders outside the 
schools often expect the use of IT to significantly increase student performance on state 
assessment measures. Typically, these groups are looking for impact to show up sooner rather 
than later. These individuals and groups (legislators, state policy makers, and even members of 
the community at large) neither expect nor respect evaluation data based solely on illustrative 
vignettes. On the other hand, those inside the system— teachers and administrators— have little 
respect (but lots of fear) for test data, preferring instead rich description of changes in learning 
opportunities and higher levels of student engagement. Teachers often want information on best 
practices. Any evaluation system must be able to provide a diversity of performance information 
that each audience will accept and respect. 

These conditions serve as a context for a closer examination of how districts design and implement 
evaluation of IT initiatives. Before moving to that examination, however, I will briefly describe the major 
features of the IT evaluation system we advocate. 

IT Evaluation System Features 

Our technical assistance to districts with respect to IT evaluation had two objectives: 1) produce 
information about implementation and impact that will inform the refinement of the IT plan and its 
implementation, and 2) enhance the capacity of district personnel to collect, analyze, and use data to 
inform their decisions about technology applications. The system design is intended to be rigorous yet 
simple enough to be incorporated into the day-to-day work of school leaders and scalable to 
accommodate ongoing changes and expansion. The system focuses on five components: 

Multiple Indicators and Measures 

We advocate the use of five types of performance measures to guide data collection. 

1 . Outcomes. Measures of results, such as student learning, equity of access to technology, and 
quality and quantity of technology-rich student learning opportunities provided by teachers. 

2. Output. Measures of products and services provided, such as the number of teachers trained, 
number of curriculum guides with embedded technology skills, and number of classrooms 
connected to school and district telecommunication networks. 

3. Input. Measures of resource allocation and use, such as the ratio of students to computers, 
resources allocated to education, training, and support activities, and allocation of technology 
resources across schools. 

4. Productivity. Measures of work performance, such as time to install school networks, hours 
of training required to prepare teachers to use technology in their classrooms, and mean time 
to respond to requests for technical assistance. 

5. Demand. Measures of potential market, such as the number of requests for pedagogical and 
technical assistance, number of teachers needing specific types of training, and number of 
classrooms with obsolete equipment. 

Each of these five types of indicators can be used to describe conditions or performance at the district 
and school levels. Of course, an input indicator at one level can be an output indicator at another. 

For example, the number and percent of teachers requesting technical support may be a demand 
indicator for the district technology coordinator and an impact indicator for those responsible for 
providing professional development. Like Einstein’s perspective on time, what one sees is a function 
of where one stands in the overall system. Appendix A provides examples of the five types of 
indicators as applied to IT applications in teaching and learning. The system can address indicators 
for administrative and management applications as well. 
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District decision-makers must achieve a consensus on what information they will need to support 
their decisions. Which objectives and initiatives are most essential to the whole enterprise? What 
questions do various stakeholders have? We stress a focus on outcomes rather than on the completion 
of activities. For example, what changes have taken place in teachers’ classroom practices rather than 
how many teachers have participated in how many workshops? 

Balanced Attention to Implementation and Impact 

The evaluation design addresses both implementation and impact. Because plans seldom get 
implemented as written, districts need to watch implementation carefully. Implementation 
monitoring typically addresses these questions such as: 

1. Were tasks completed as designed? Were implementation timelines met? On budget? 

2. What barriers were encountered during implementation? How were they addressed? 

3. What changes were implemented? Why? To what effect? 

4. What are the implications of these mid-course corrections for redesign? 

Implementation monitoring usually is addressed to all five types of indicators, with the possible 
exception of outcomes. 

As part of the IT planning process, districts identify specific indicators and measures for each priority 
objective. They identify these indicators by reviewing literature and research as well as crafting their 
own indicators. Jones et. ah (1995), for example, have identified key indicators for engaged learning 
and for high technology performance and proposed a process for using those indicators to assess the 
effectiveness of individual technologies and technology-enhanced programs in support of engaged 
learning. Her approach identifies proximal indicators based on explicit or implicit theories of action 
supporting the use of technology applications in teaching and learning. Others have also suggested 
similar variables and impact indicators (Dwyer, 1994; NSSE, 1996; Means & Olson, 1996; Quinones 
& Kirshstein, 1998). 

Multiple Data Collection and Reporting Strategies 

The system calls for collecting both quantitative and qualitative data. A variety of data collection 
instruments and procedures are employed: journals, critical event logs, documentation, group 
interviews, and focus groups are employed along with quantitative measures. The system employs 
multiple methods and instruments for helping everyone to watch performance and contribute mid- 
course adjustments. All data collection is addressed to the indicators and performance measures 
established in the IT plan. Data are analyzed and reported in real time and in multiple formats. 

Use of IT for Evaluation 

The system employs databases, preferably electronic, for collecting, organizing, and disseminating 
data and information in diverse forms to serve multiple audiences. For example, the system provides 
guidance in developing databases of best practices in several categories— teaching and learning, 
communications, and administration. These databases address what is working and why and what is 
not working and why. Databases can also be used to set up discussions on various instructional and 
administrative strategies; even chat rooms among teachers and principals are very useful. The intent is 
to make information available to all using the district’s telecommunications infrastructure as well as 
more traditional means. Kaplan & Norton (1996), for example, suggest the use of electronic databases 
and “balanced scorecards” to automatically alert and “informate” (Zuboff, 1988) those who need to 
know about specific data, such as a principal tracking a new program’s performance or a teacher 
watching her students’ learning on a key learning standard. In districts with more advanced IT 
systems, Web-based databases can capture, organize, and make widely accessible information on the 
scores of small, immediate adjustments made during implementation. These systems can provide up- 
to-date information on the status of each indicator. 
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Enhance District Capacity 

Since a principal objective is to enhance the district’s evaluation capacity, major attention is given to 
helping the district establish an organizational capability to collect, analyze, and use the data. Unless 
district administrators and teachers can make sense of what is going on, the system will be of little 
use. Part of the technical assistance, therefore, is devoted to nurturing a disposition to use the system 
to produce useful information. This part of the work has benefits in addressing larger district 
capacities that go beyond evaluating information technology applications. For this reason, we 
advocate that the district establish a committee to oversee the IT evaluation of staff responsible for 
such functions as curriculum and instruction, staff development, and assessment as well as 
technology. 

Rather then centralizing the responsibility for IT evaluation in the technology department or with 
technical specialists, we advocate assigning data collection on indicators to those 
personnel/organizational units accountable for accomplishing the objectives related to those 
indicators. Thus, principals and staff development providers are responsible for providing education, 
training, and support to teachers, so they are accountable for collecting information on the indicators. 



The evaluation system and its use are still evolving; the design and process are highly customized in each 
district. The system is based on best evaluation practice with a heavy emphasis on helping districts 
incorporate into their operations a habit of systematically watching indicators that matter to them and 
those they serve, directly and indirectly. Our experience is that few districts make a commitment to 
address all components and tasks. 



Findings 

Based on work to date in districts and schools of varying size, we have identified several preliminary 
findings. These findings are derived from participant observations, structured and unstructured interviews, 
focus groups, and document reviews conducted over the last three years. Most of what we have learned 
about IT evaluation thus far can be organized into three areas: IT planning, IT evaluation design, and IT 
evaluation implementation. 

IT Planning 

It was often possible to trace a substantial part of the problems with IT evaluation to faulty IT planning 
and design. It appears that the planning process predisposed school practitioners as well as other 
stakeholders to see IT as a separable system requiring isolated and unique attention. Moreover, although 
most planning models mention evaluation, it was seldom addressed as an integral part of the plan. In 
many cases, the design of the plan actually precluded serious attention to evaluation. While this variable 
will be examined more carefully in future research, we have at this stage identified three major reasons 
for this problem. 

Multiple and unconnected plans. Most districts had two or more plans that were not well integrated. 
For example, it was not uncommon for a district to have a strategic plan and a technology plan with 
few linkages. Often the strategic plan had a strategic initiative addressing IT, but in most cases the 
integration of the IT plan with the strategic plan was weak or non-existent. 

Inadequate IT Plans. IT plans were weak in several areas; an inordinate focus on the IT 
infrastructure, a narrow vision, and inadequate attention to staff and organizational development were 
most prominent. Most important for this research, plans often failed to delineate objectives 




5 



addressing improvements in teaching and learning. It was unusual to find a plan that provided 
objectives and action plans related to the redesign of student learning opportunities with technology. 

Shared Expectations of Success. IT planners often failed to transform their technology vision 
statements into a shared definition of what they considered success. Thus, merely installing 
hardware, software, and networks were viewed by some as success while others looked for objectives 
relating to student learning. Even those who focused on student learning often were not clear about 
the specific student learning results they expected. 

IT Evaluation Designs 

We identified three specific problems relating to IT evaluation designs. 

Mindsets regarding IT. What gets evaluated was determined by the images that planners formed 
about IT applications. Despite the evidence that IT can support a substantial transformation of 
learning opportunities and environments (Collins, 1991; Madian, 1991; Jones et. al., 1995; Means & 
Olson, 1995; Salomon & Perkins, 1996), most IT plans addressed relatively mundane applications 
that did not challenge the prevailing pedagogy. The use of such applications was most prevalent in 
urban classrooms where the technology was addressed to low-level skill development at the expense 
of more holistic and engaging learning opportunities. 

Dispositions toward IT evaluation . A key factor impacting the scope and quality of IT evaluation 
systems was the way that key decision-makers in the district or school viewed IT and its potential 
impact on their mission and strategies. Those who saw IT as a separate entity tended to see IT 
evaluation that way as well. Despite the age-old admonition to build in evaluation, it was either 
ignored or bolted on to the plan. In a small minority of districts where IT was integrated into the 
district strategic plan, the integration of evaluation of its impact was integrated as well. Even in such 
circumstances, however, there was no guarantee that IT evaluation would be accomplished well or at 
all. In some districts, for example, administrators believed that evaluation consisted of determining 
whether reading scores had improved as a result of IT applications. 

Narrow indicators and measures. Districts tended to address indicators dealing with building the IT 
infrastructure or with assessing impact on student achievement as measured by standardized tests. 
None of the districts developed indicators relating to such variables as changes in student learning 
opportunities, engagement in learning, and attention to higher order or complex learning. In short, 
evaluation plans seldom if ever focused on developing a deep understanding of what impact IT was 
having on what students experience. There was little or no attention to examining the chain of 
evidence relating to a theory of action regarding IT’s impact on learning. Districts tended to rely on 
anecdotal data, but without the rigor or the comprehensiveness that would constitute high quality data 
support. Districts’ IT evaluation designs failed most in understanding what Salomon (1991) calls the 
systemic nature of their IT initiatives. Most IT applications consisted of a rich soup of highly 
interdependent interventions that impacted one another in highly complex ways. 

Evaluation Management 

Most districts treated IT evaluation as a compartmentalized responsibility assigned to the IT department 
or selected IT staff. Data collection was often conducted by the IT staff or by the district central office 
evaluation staff. Other district staff members, such as those dealing with curriculum development, staff 
development, and assessment, seldom engaged in IT evaluation or saw it as their responsibility. They 
saw the data on IT’s effectiveness as peripheral to their own missions. IT staff members were usually so 
burdened with the technical tasks of installing and maintaining the infrastructure that time for evaluation 
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was limited. Moreover, their focus was usually on the technical aspects of the system as distinguished 
from its classroom applications. 

Implications for Practice, Policy, and Research 

The objective of our ongoing work is to better understand how districts and schools plan, design, and 
conduct evaluation of their IT initiatives in order to inform improvement and redesign. Through intensive 
case studies and collaborative learning and work with districts and schools, we have formed several 
insights about the facilitators and impediments to IT evaluation. The following implications are based on 
those findings. 

Despite the fact that many districts were well into their second or third cycles of IT planning, few had a 
comprehensive evaluation in place to provide information on implementation and impact. IT evaluation 
received a low priority from most districts, with few having any system in place. In those districts that 
did undertake IT evaluation, the task was often either poorly designed or not managed well. IT evaluation 
designs often focused on the wrong questions, employed inadequate designs, and failed to employ 
systematic analysis and ongoing reporting of findings to decision-makers and stakeholders. Districts 
failed to watch in any systematic way the way that IT actually was implemented and what effects it had. 
There was little documentation of the interventions being implemented. Results were neither clearly 
specified nor focused on student learning. There was no process for building a shared commitment to 
results. Responsibility for evaluation was often isolated within districts’ IT departments. 



IT Planning 

In many cases problems with IT evaluation originate in flawed IT planning. With the early stages of IT 
planning so focused on installing equipment and networks, the lack of strategic thinking focused on 
transforming teaching and learning is not surprising. Most districts use the technology to automate or 
augment their current curriculum and pedagogy rather than think about a new one. Plans therefore give 
little attention to redesigning learning opportunities, learning environments, and pedagogy. IT evaluation, 
therefore, gives little or no attention to these areas. 

While districts often linked their strategic plans with their IT plans, such linkages may not be sufficient to 
accomplish integration. Instead, it may be necessary to incorporate IT planning directly into the district’s 
strategic or improvement plan. This one plan would provide the overarching framework for improvement 
and delineate how IT will contribute to that improvement. While specific initiatives such as the 
installation of the telecommunications network might require ongoing implementation planning, from a 
strategic perspective there may be no need for a separate strategic technology plan. Instead, all 
technology planning should be focused on what technology applications are required to advance the 
district’s annual performance goals in support of its reform initiatives. With the principal focus on the 
district’s overarching improvement plan, IT indicators of success linked to strategic objectives could 
focus the IT evaluation. 

In large districts where there is likely to be a chief technology officer who is at cabinet level, there 
appears to be a greater chance for integrating the IT plan with the district’s strategic plan. This is not the 
case in smaller districts where the “chief technology officer” is often a technical specialist without 
program and policy perspectives. Integration of IT planning into the district improvement planning and 
plan might produce another benefit: the involvement of the district leadership in understanding how IT 
can contribute to the district’s priorities for improvement. Because district superintendents typically lead 
strategic planning efforts but seldom participate on technology planning committees, merging the two 
planning efforts would serve to engage district leaders in assuring technology integration at all levels. 
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Federal and state requirements as well as district practices that require or promote separate planning and 
plans may work against the desired results. IT integration into student learning opportunities and 
environments will likely be impeded by inadequate integration at every other level. 

Further investigation of IT planning and its attention to evaluation might focus on these questions: 

• How is IT planning accomplished? 

• What is the relationship between strategic or improvement planning and IT planning? 

• What data on implementation and impact are employed in the planning process? What is the 
quality of that data? 

• What is the role of the district superintendent in IT planning? What roles do curriculum directors, 
staff development coordinators, and other central office staff have in IT planning? 

IT Evaluation Design 

IT design has not, in most cases, provoked a systematic and rigorous approach to evaluation, even in 
districts that are undertaking a serious redesign of student learning opportunities and learning 
environments. Faulty evaluation design often precludes the likelihood of collecting any useful 
information. Several factors appear to be important; prominent among them is selecting variables and 
indicators and employing appropriate measures and methods. 

Selecting appropriate variables and tracking relevant indicators poses the most challenge for districts. 
Practitioners recognize what researchers lament: Distilling out the unique contribution of IT in a complex 
intervention is a formidable task. Rather than attempt such a distillation, Salomon (1991) and others 
(Pappert, 1987; Liu, Macmillan, & Timmons, 1998) argue that we need, at this early stage of our use of 
technology in schools, to embrace the complexity and seek to document its dimensions. 

As Hativa and Lesgold (1996) point out, the whole learning environment changes, making it difficult to 
undertake analytical studies, particularly in schools. IT applications need to be viewed as complex 
bundles of changes. It is unlikely that districts have enough regression analysis tools to accomplish the 
distillation required for analytic work with respect to most of these variables. Districts implement a 
bundle of interventions when they introduce technology in support of teaching and learning. Thus, they 
need to examine the interface between IT and the curriculum materials, content, pedagogy professional 
development, assessment, and the learning environment. In many instances it is too soon for analytic 
approaches that attempt to isolate the unique contribution of the technology on student performance, 
particularly when measured by standardized tests. There is a need for a balanced attention to analytic and 
systemic approaches (Salomon, 1991). Rich description is needed. 

Rather than attempting to distill out the unique contribution of IT to a specific initiative or even to a 
specific learning outcome, it might be better to use more holistic, qualitative data to rich descriptions of 
what is happening in classrooms and with learning opportunities. In many cases information technology’s 
real impact may be in areas that are not under the analytic spotlight. It will take a systemic approach to 
illuminate the issues and their interactions. A system that collects rich/thick description may be more 
appropriate and is the role that practitioners may be able to perform best. The challenge is to create simple 
systems for teachers to collect data on the learning partnerships that Salomon says may be an indicator of 
high-performance learning. 

IT’s real impact may be in areas receiving little district attention. Instead of asking, “Do computers 
work,” districts need to ask, “What specific applications, under what circumstances, appear to contribute 
most to increased student learning?” With such a system in place, the district can be more precise in 
addressing questions about the broad impact of IT investments as well as on more specific questions 
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regarding specific outcomes. Districts need rich description as well as quantitative data on a diverse set of 
variables, most often focused where the real impact may be — on the nature of the learning process itself 

Districts need to follow standards for assuring quality data collection and analysis. Decision-makers must 
have confidence in the reliability and validity of the data. They must create simple ways to collect robust 
qualitative data. These can best be captured through journals and critical event logs. These methods are 
resource intensive but valuable. This is the rich descriptive data that captures the complexity of the 
applications. 

Policy makers at federal and state levels need to promote these systemic approaches by collecting 
qualitative as well as quantitative data from districts and helping them to analyze it. State education 
agencies and colleges and universities could help as well with collecting and disseminating descriptions 
of practices and their results in classrooms. 

State education agencies might provide technical assistance to district staff in designing and 
implementing IT evaluation systems. In higher education, the Flashlight Program, for example, offers 
help to colleges and universities in evaluating their IT applications ((Ehrmann, 1999). 

Further investigation of IT evaluation might focus on these questions: 

• What do districts evaluate with respect to technology applications? 

• How do districts conduct IT evaluation? What data collection systems are in place? 

• What systems do districts establish for documentation and sharing? 

• What roles do curriculum directors, staff development coordinators, and other central office staff 
play in IT evaluation? 

IT Evaluation Implementation 

Three things would improve the implementation of IT evaluation in districts and schools: 1) increasing 
the frequency of data collection, 2) decentralizing responsibility for IT evaluation throughout the district, 
and 3) increasing the dissemination of evaluation information throughout the district and to the 
community. 

Data collection needs to be ongoing and linked to key indicators. Evaluation needs to provide regular, 
real-time, ongoing data flowing throughout the organization and beyond about what works and what 
changes are being made. Rather than waiting until the end of the year or the end of a project, data must 
be provided on an ongoing basis. Decision-makers must use the data in the real-time flow of decisions. 
The intent of the evaluation should be to encourage and support staff in watching implementation in a 
more systematic and sophisticated manner. This is consistent with an action research orientation. 

The locus of responsibility for IT evaluation appears to be a factor in determining whether and how well it 
is accomplished. The organizational disposition to compartmentalize functions works against an 
integrated approach to IT evaluation. Rather than assign responsibility to the IT technical staff, therefore, 
it might be more productive to decentralize it. 

Districts might establish a committee to continue the work of the group that provided oversight to the 
development of the technology plan. The committee might be expanded to include more school personnel 
who are close to the implementation of selected technology initiatives. More directly, oversight of the 
data collection process should be assigned to those personnel/organizational units accountable for 
accomplishing the objectives related to those indicators. When specific staff members are accountable for 
certain outcomes, they will have a built-in self-interest in the scope and quality of the data they require. If 
this alignment of need and use is not established, communication is defined as a problem of dissemination 
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— convincing various people to access data they are not convinced they need. The committee might meet 
quarterly and use e-mail to meet electronically as needed to review data and to comment on needed mid- 
course corrections. 

The assessment process needs to be simultaneously top-down and bottom-up, gathering information from 
the operating system about what is happening in schools with respect to the objectives, what new or 
enhanced interventions are needed to obtain the results, and what new or unanticipated outcomes and 
results are being realized. 

Districts would benefit substantially from an increased attention to organizational learning and knowledge 
management with respect to IT. To accomplish this, some districts use simple, electronic Web-based 
databases for collecting, organizing, and disseminating data and information in diverse forms. One 
example that is quite useful is to develop databases of best practices in several categories: teaching and 
learning, communications, and administration. These databases can address what is working and why and 
what is not working and why. Databases are also used to set up discussions on various instructional and 
administrative strategies; chat rooms among teachers and principals are very useful. Prepare a public 
database for communicating results to the larger community. 

Districts need to prepare a portfolio of program assessments and regularly updated reports of the specific 
technology initiatives implemented in response to specific district and school reform initiatives. These 
reports can serve as a component of a tracking mechanism for monitoring implementation of specific 
elements of the technology plan. 

Further research to enlarge our understanding should address these questions: 

• How is responsibility for IT evaluation assigned? Is there a relationship between the locus of 
responsibility for IT evaluation and its implementation? Is there a relationship between the locus 
of responsibility for IT evaluation and its use in decision making? 

• What roles do curriculum directors, staff development coordinators, and other central office staff 
have in IT evaluation? 

• How do districts communicate evaluation information to key decision-makers? 



Improving IT evaluation is important not only because of the substantial investments districts are making 
in information technology but also because: a) we are at the early stages of what potentially is a learning 
revolution, b) the public needs to know, and c) our understanding of IT’s impact is so limited. It is 
important for researchers to connect to what is actually happening in practice, not just in pilot projects and 
demonstration programs but in schools and classrooms throughout the country. 

Public support for continuing investments in technology are likely to wane unless districts and schools 
can communicate effectively about the impact of IT on teaching and learning. This will require a quantity 
and quality of information that is usually available from pilot or demonstration projects with research- 
based designs. The continuing challenge is to help schools set up data collection systems and help them 
make sense of the data for ongoing improvement and redesign. The research community cannot ignore 
the sorry state of IT evaluation in districts and schools. Future efforts must focus on: 

• Developing a better understanding of district IT evaluation through more focused research. 

• Helping practitioners through partnerships with researchers not only to conduct the evaluation but 
also to build their capacity to collect better data for their own learning. 
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Appendix A 
Examples of Indicators 



Indicator 


Teaching & Learning 


Impact 


• quantity and quality of student writing 

• % of teachers using technology applications in their day-to-day 
teaching 


Output 


• # of workshops provided to teachers, administrators, and staff 

• #/% of teachers, administrators, and staff trained 

• % of time network is in service 


Input 


• #/% of classrooms networked 

• student/computer ratio 


Productivity 


• # of training hours needed to achieve a specific teacher competency, 
such as use of the Internet 

• time devoted to taking and reporting attendance 


Demand 


• # of teachers and administrators requesting training and support 

• # of parents requesting training and support 
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